Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avro schema validation #105

Merged

Conversation

Big-Vi
Copy link
Contributor

@Big-Vi Big-Vi commented Jul 26, 2023

switch msg.(type) {
case []byte:
msgBytes = msg.([]byte)
if err := json.Unmarshal(msgBytes, &message); err != nil {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

myData := []byte({"username": "john", "age": 30})
When json unmarshal the above data, int type is converted to float64 thus causing schema validation error. So when user is defining avro schema, schema can't have int type. It should be double rather than int.

https://github.com/hamba/avro#types-conversions

Screenshot 2023-07-26 at 1 18 08 PM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any workaround to solve this? otherwise we should mention it on the UI as well

Copy link
Contributor Author

@Big-Vi Big-Vi Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any workaround to solve this? otherwise we should mention it on the UI as well

I can think of the below solution. But this solution converts int to int64. Still we need to add note in the UI.
So i reckon leave it as it is and add note in the UI.

https://go.dev/play/p/SkG6qlolz1B

myData := `{"username": "John", "age": 30}`
var m map[string]interface{}

dec := json.NewDecoder(strings.NewReader(myData))
dec.UseNumber()

err := dec.Decode(&m)
if err != nil {
     log.Fatal(err)
}

m["age"], err = m["age"].(json.Number).Int64()
if err != nil {
	log.Fatal(err)
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok totally agree

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this thing happening on other SDKs as well or only in Go? if no so let's add a comment about it in the readme file of Go sdk it is enough

switch msg.(type) {
case []byte:
msgBytes = msg.([]byte)
if err := json.Unmarshal(msgBytes, &message); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any workaround to solve this? otherwise we should mention it on the UI as well

return nil, memphisError(err)
}

default:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In protobuf we allow users to send map[string]interface{} in order people won't have to hole the proto file locally, is it possible here as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I've added it now.

@idanasulin2706 idanasulin2706 merged commit dda267f into superstreamlabs:master Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants